torch.distributed.barrier has an interesting behavior that may cause our tokenizer (and maybe other parts of the toolkit) to explode. Any call to torch.distributed.barrier will automatically let any other process having encountered torch.distributed.barrier to continue, even if it's a different one. This means that if we have a torch.distributed.barrier in a if statement that is only encountered by the main process, it can just lift the barrier of the other process. Which is what may happen with our tokenizer... An example is the following code:

    if if_main_process():
        import time
        time.sleep(5)
        print("Barrier main")
        torch.distributed.barrier()
        time.sleep(5)
        print("End main process")
    else:
        print("First barrier second process")
        torch.distributed.barrier()
        print("End first barrier second process") ```