در حال ران شبکه عصبی کانولوشنی در فضای ابری هستم با این ارور در زمان ران شبکه مواجه میشوم

---

شرح پروژه

2024-09-22 16:36:39.607420: W external/local_tsl/tsl/framework/bfc_allocator.cc:485] Allocator (GPU_0_bfc) ran out of memory trying to allocate 664.67MiB (rounded to 696960000)requested by op _EagerConst
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. 
Current allocation summary follows.
Current allocation summary follows.
2024-09-22 16:36:39.607503: I external/local_tsl/tsl/framework/bfc_allocator.cc:1039] BFCAllocator dump for GPU_0_bfc
2024-09-22 16:36:39.607527: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (256):     Total Chunks: 57, Chunks in use: 57. 14.2KiB allocated for chunks. 14.2KiB in use in bin. 6.1KiB client-requested in use in bin.
2024-09-22 16:36:39.607545: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (512):  7ff0521ddc00 of size 24576 next 27
2024-09-22 16:36:39.609082: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] Free  at 7ff0521e3c00 of size 135168 next 113
2024-09-22 16:36:39.609096: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7ff052204c00 of size 61440 next 112
2024-09-22 16:36:39.609110: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7ff052213c00 of size 61440 next 78
2024-09-22 16:36:39.609124: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] Free  at 7ff052222c00 of size 5984256 next 110
2024-09-22 16:36:39.609138: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7ff0527d7c00 of size 2097152 next 109
2024-09-22 16:36:39.609152: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] Free  at 7ff0529d7c00 of size 27000832 next 107
2024-09-22 16:36:39.609224: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7ff054397c00 of size 11796480 next 106
2024-09-22 16:36:39.609239: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] Free  at 7ff054ed7c00 of size 434471936 next 18446744073709551615
2024-09-22 16:36:39.609256: I external/local_tsl/tsl/framework/bfc_allocator.cc:1100]      Summary of in-use Chunks by size: 
2024-09-22 16:36:39.609283: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 57 Chunks of size 256 totalling 14.2KiB
2024-09-22 16:36:39.609301: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 1280 totalling 1.2KiB
2024-09-22 16:36:39.609321: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 2048 totalling 4.0KiB
2024-09-22 16:36:39.609340: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 4 Chunks of size 4096 totalling 16.0KiB
2024-09-22 16:36:39.609356: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 12288 totalling 24.0KiB
2024-09-22 16:36:39.609371: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 24576 totalling 48.0KiB
2024-09-22 16:36:39.609386: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 61440 totalling 120.0KiB
2024-09-22 16:36:39.609401: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 2097152 totalling 4.00MiB
2024-09-22 16:36:39.609416: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 11796480 totalling 22.50MiB
2024-09-22 16:36:39.609431: I external/local_tsl/tsl/framework/bfc_allocator.cc:1107] Sum Total of in-use chunks: 26.72MiB
2024-09-22 16:36:39.609452: I external/local_tsl/tsl/framework/bfc_allocator.cc:1109] Total bytes in pool: 517144576 memory_limit_: 517144576 available bytes: 0 curr_region_allocation_bytes_: 1034289152
2024-09-22 16:36:39.609477: I external/local_tsl/tsl/framework/bfc_allocator.cc:1114] Stats: 
Limit:                       517144576
InUse:                        28020224
MaxInUse:                     68960768
NumAllocs:                         270
MaxAllocSize:                 17301504
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2024-09-22 16:36:39.609499: W external/local_tsl/tsl/framework/bfc_allocator.cc:497] ***___*_*____***____________________________________________________________________________________ز

مهارت ها و تخصص های مورد نیاز

---

---