torchserve – How can i create torch server on Google Colab and use prediction

I try to create a torchserve on google colab but it took forever to load and it seems that i can’t even connect to the serve. Is this possible to create a torchserve on colab? Here is what it show when i try to do it.

Removing orphan pid file. WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. 2022-05-01T10:56:00,634 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager – Initializing plugins manager… 2022-05-01T10:56:00,775 [INFO ] main org.pytorch.serve.ModelServer – Torchserve version: 0.5.3 TS Home: /usr/local/lib/python3.8/site-packages Current directory: /content Temp directory: /tmp Number of GPUs: 0 Number of CPUs : 2 Max heap size: 3248 M Python executable: /usr/local/bin/python Config file: /content/drive/MyDrive/COferenceGerman/config.properties Inference address: http://0.0.0.0:8000 Management address: http ://0.0.0.0:8001 Metrics address: http://0.0.0.0:8002 Model Store: /content/drive/MyDrive/COferenceGerman Initial Models: droc_incremental_no_segment_distance.mar Log dir: /content/logs Metrics dir: /content/ logs Netty threads: 0 Netty client threads: 0 Default workers per model: 2 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.|http(s)?://.]
Custom python dependency for model allowed: false Metrics report format: prometheus Enable metrics API: true Workflow Store: /content/drive/MyDrive/COferenceGerman Model config: N/A 2022-05-01T10:56:00,784 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager – Loading snapshot serializer plugin… 2022-05-01T10:56:00,824 [INFO ] main org.pytorch.serve.ModelServer – Loading initial models: droc_incremental_no_segment_distance.mar 2022-05-01T10:56:13,006 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs – Adding new version 0.0.1 for model droc_incremental_no_segment_distance 2022-05-01T10:56:13,009 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs – Setting default version to 0.0.1 for model droc_incremental_no_segment_distance 2022-05-01T10:56:13,010 [INFO ] main org.pytorch.serve.wlm.ModelManager – Model droc_incremental_no_segment_distance loaded. 2022-05-01T10:56:13,011 [DEBUG] main org.pytorch.serve.wlm.ModelManager – updateModel: droc_incremental_no_segment_distance, count: 2 2022-05-01T10:56:13,032 [INFO ] main org.pytorch.serve.ModelServer – Initialize Inference server with: EpollServerSocketChannel. 2022-05-01T10:56:13,039 [DEBUG] W-9000-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle – Worker cmdline: [python3, /usr/local/lib/python3.8/site-packages/ts/model_service_worker.py, –sock-type, unix, –sock-name, /tmp/.ts.sock.9000]
2022-05-01T10:56:13,051 [DEBUG] W-9001-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle – Worker cmdline: [python3, /usr/local/lib/python3.8/site-packages/ts/model_service_worker.py, –sock-type, unix, –sock-name, /tmp/.ts.sock.9001]
2022-05-01T10:56:13,318 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – Traceback (most recent call last): 2022-05-01T10:56:13,327 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – File “/usr/local/lib/python3.8/site-packages/ts/model_service_worker.py”, line 17, in 2022-05-01T10:56:13,329 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – from ts.model_loader import ModelLoaderFactory 2022-05-01T10:56:13,331 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – File “/usr/local/lib/python3.8/site-packages/ts/model_loader.py”, line 16, in 2022-05-01T10:56:13,333 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – from ts.service import Service 2022-05-01T10:56:13,333 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – File “/usr/local/lib/python3.8/site-packages/ts/service.py”, line 14, in 2022-05-01T10:56:13,339 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – from ts.protocol.otf_message_handler import create_predict_response 2022-05-01T10:56:13,340 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – File “/usr/local/lib/python3.8/site-packages/ts/protocol/otf_message_handler.py”, line 15, in 2022-05-01T10:56:13,341 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – import torch 2022-05-01T10:56:13,347 [INFO ] main org.pytorch.serve.ModelServer – Inference API bind to: http://0.0.0.0:8000 2022-05-01T10:56:13,349 [INFO ] main org.pytorch.serve.ModelServer – Initialize Management server with: EpollServerSocketChannel. 2022-05-01T10:56:13,350 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – ModuleNotFoundError: No module named ‘torch’ 2022-05-01T10:56:13,367 [INFO ] main org.pytorch.serve.ModelServer – Management API bind to: http://0.0.0.0:8001 2022-05-01T10:56:13,367 [INFO ] main org.pytorch.serve.ModelServer – Initialize Metrics server with: EpollServerSocketChannel. 2022-05-01T10:56:13,369 [INFO ] main org.pytorch.serve.ModelServer – Metrics API bind to: http://0.0.0.0:8002 2022-05-01T10:56:13,373 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – Traceback (most recent call last): 2022-05-01T10:56:13,377 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – File “/usr/local/lib/python3.8/site-packages/ts/model_service_worker.py”, line 17, in 2022-05-01T10:56:13,377 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – from ts.model_loader import ModelLoaderFactory 2022-05-01T10:56:13,378 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – File “/usr/local/lib/python3.8/site-packages/ts/model_loader.py”, line 16, in 2022-05-01T10:56:13,378 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – from ts.service import Service 2022-05-01T10:56:13,378 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – File “/usr/local/lib/python3.8/site-packages/ts/service.py”, line 14, in 2022-05-01T10:56:13,378 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – from ts.protocol.otf_message_handler import create_predict_response 2022-05-01T10:56:13,379 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – File “/usr/local/lib/python3.8/site-packages/ts/protocol/otf_message_handler.py”, line 15, in 2022-05-01T10:56:13,379 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – import torch 2022-05-01T10:56:13,379 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr MODEL_LOG – ModuleNotFoundError: No module named ‘torch’ 2022-05-01T10:56:13,400 [INFO ] W-9001-droc_incremental_no_segment_distance_0.0.1-stderr org.pytorch.serve.wlm.WorkerLifeCycle – Stopped Scanner – W-9001-droc_incremental_no_segment_distance_0.0.1-stderr 2022-05-01T10:56:13,403 [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle – terminateIOStreams() threadName=W-9001-droc_incremental_no_segment_distance_0.0.1-stderr 2022-05-01T10:56:13 [INFO ] W-9001-droc_incremental_no_segment_distance_0.0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle – Stopped Scanner – W-9001-droc_incremental_no_segment_distance_0.0.1-stdout 2022-05-01T10:56:13, [WARN ] W-9001-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle – terminateIOStreams() threadName=W-9001-droc_incremental_no_segment_distance_0.0.1-stdout 2022-05-01T1028:56:13 [INFO ] W-9000-droc_incremental_no_segment_distance_0.0.1-stderr org.pytorch.serve.wlm.WorkerLifeCycle – Stopped Scanner – W-9000-droc_incremental_no_segment_distance_0.0.1-stderr 2022-05-01T10:56:13,428 [INFO ] W-9000-droc_incremental_no_segment_distance_0.0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle – Stopped Scanner – W-9000-droc_incremental_no_segment_distance_0.0.1-stdout 2022-05-01T10:56:13,4 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle – terminateIOStreams() threadName=W-9000-droc_incremental_no_segment_distance_0.0.1-stderr 2022-05-01430:56:13 [WARN ] W-9000-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle – terminateIOStreams() threadName=W-9000-droc_incremental_no_segment_distance_0.0.1-stdout 2022-05-01T10:56:13 [ERROR] W-9000-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerThread – Backend worker error org.pytorch.serve.wlm.WorkerInitializationException: Backend stream closed. at org.pytorch.serve.wlm.WorkerLifeCycle.startWorker(WorkerLifeCycle.java:139) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.connect(WorkerThread.java:292) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:179) [model-server.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2022-05-01T10:56:13,408 [ERROR] W-9001-droc_incremental_no_segment_distance_0.0.1 org.pytorch.serve.wlm.WorkerThread – Backend worker error org.pytorch.serve.wlm.WorkerInitializationException: Backend stream closed. at org.pytorch.serve.wlm.WorkerLifeCycle.startWorker(WorkerLifeCycle.java:139) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.connect(WorkerThread.java:292) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:179) [model-server.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Model server started. 2022-05-01T10:56:13,759 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector – worker pid is not available yet. 2022-05-01T10:56:13,833 [INFO ] pool-3-thread-1 TS_METRICS – CPUUtilization.Percent:100.0|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,835 [INFO ] pool-3-thread-1 TS_METRICS – DiskAvailable.Gigabytes:66.32177734375|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,837 [INFO ] pool-3-thread-1 TS_METRICS – DiskUsage.Gigabytes:41.377681732177734|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,838 [INFO ] pool-3-thread-1 TS_METRICS – DiskUtilization.Percent:38.4|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,839 [INFO ] pool-3-thread-1 TS_METRICS – MemoryAvailable.Megabytes:11904.3984375|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,840 [INFO ] pool-3-thread-1 TS_METRICS – MemoryUsed.Megabytes:794.34375|#Level:Host|#hostname:649a8b344986,timestamp:1651402573 2022-05-01T10:56:13,841 [INFO ] pool-3-thread-1 TS_METRICS – MemoryUtilization.Percent:8.3|#Level:Host|#hostname:649a8b344986,timestamp:1651402573

Any idea how can i fix this on colab? It work normal on my pc but i need it on colab.

Leave a Comment