Cloudian Hyperstore Errors


#1

I configured Cloudian Hyperstore with Starter Edition. When I tried to run migrate task from Windows to Cloudian Hyperstore these errors were logged.
What cause these errors?

2018-02-19 23:22:09.277 (GMT+09:00) ERROR: FAILED to Migrate win://server1.worknetwork.org/E/myfile.jpg to s3cloudian://gateway.worknetwork.org/s3-region1.office.worknetwork.org/mybucket/
2018-02-19 23:22:09.277 (GMT+09:00) [0] ERR_DMPROTOCOLAGENT_DECODEASYNCMESSAGE_FAILED [] [migrate win://server1.worknetwork.org/E/myfile.jpg to dmagent://gateway.worknetwork.org/s3cloudian%3A//gateway.worknetwork.org/s3-region1.office.worknetwork.org/mybucket/] dmprotocolagent.cpp:1831
2018-02-19 23:22:09.277 (GMT+09:00)  [1] ERR_DMAGENT_MIGRATE_FAILED [] [] dmagent.cpp:3337
2018-02-19 23:22:09.277 (GMT+09:00)   [2] ERR_DMAGENT_MIGRATEDATATODESTINATION_FAILED [] [] dmagent.cpp:3811
2018-02-19 23:22:09.277 (GMT+09:00)    [3] ERR_STREAMAGENT_OPEN_FAILED [] [] dmstreamagent.cpp:2312
2018-02-19 23:22:09.277 (GMT+09:00)     [4] ERR_STREAMAGENT_OPENCONNECTION_FAILED [ERR_ADD_OPEN_FAILED] [dmagent://gateway.worknetwork.org/s3cloudian%3A//gateway.worknetwork.org/s3-region1.office.worknetwork.org/mybucket/server1.worknetwork.org_E/3A/3B/3A3B1234XMZU6P6.mwi] dmstreamagent.cpp:288
2018-02-19 23:22:09.277 (GMT+09:00)      [5] ERR_SSLCLIENT_OPEN_FAILED [] [] sslclient.cpp:206
2018-02-19 23:22:09.277 (GMT+09:00)       [6] ERR_DMSOCKETUTIL_GETROUNDROBINCONNECTEDSOCKET_FAILED [ERR_ADD_ALL_NODES_BAD] [gateway.worknetwork.org] dmsocketutil.cpp:694
2018-02-19 23:22:09.277 (GMT+09:00)        [7] ERR_DMSOCKETUTIL_GETROUNDROBINCONNECTEDSOCKET_FAILED [ERR_ADD_NO_CONNECTION_AVAILABLE] [] dmsocketutil.cpp:646
2018-02-19 23:22:09.277 (GMT+09:00)         [8] ERR_DMSOCKETUTIL_GETROUNDROBINCONNECTEDSOCKET_FAILED [] [host [gateway.worknetwork.org with ip 192.168.1.100], port 4604] dmsocketutil.cpp:635
2018-02-19 23:22:09.277 (GMT+09:00)          [9] ERR_DMSOCKETUTIL_GETNEWCONNECTEDSOCKET_FAILED [ERR_ADD_CONNECT_FAILED] [WSAETIMEDOUT] dmsocketutil.cpp:778

#2

It looks like you are having network connection problems to the gateway “gateway.worknetwork.org

This can be seen from the error stack here.

[8] ERR_DMSOCKETUTIL_GETROUNDROBINCONNECTEDSOCKET_FAILED [] [host [gateway.worknetwork.org with ip 192.168.1.100], port 4604] dmsocketutil.cpp:635
    [9] ERR_DMSOCKETUTIL_GETNEWCONNECTEDSOCKET_FAILED [ERR_ADD_CONNECT_FAILED] [WSAETIMEDOUT] dmsocketutil.cpp:778

Make sure your machines are properly connected to the network, firewalls, etc.


#3

Oops yes there was a separate firewall box.

Now I am getting this error. Does this mean Moonwalk could not migrate any data to the target? Please let me know what this means.

ERROR processing remote stream request: 
[0] ERR_DMSTREAMAGENTSERVER_DECODEMESSAGE_FAILED [ERR_ADD_REMOTED_STREAM_OPERATION_FAILED] [xxxxx] dmstreamagentserver.cpp:2336 
 [1] ERR_DMSTREAMAGENTSERVER_FLUSH_FAILED [] dmstreamagentserver.cpp:715
  [2] ERR_DMSTREAMAGENTSERVER_ASYNCWRITE_FAILED [] dmstreamagentserver.cpp:646
   [3] ERR_DMSTREAMAMAZONS3_WRITE_FAILED [] dmstreamamazons3.cpp:793
    [4] ERR_DMSTREAMAMAZONS3_WRITEDATATOBLOCK_FAILED [] dmstreamamazons3.cpp:1141
     [5] ERR_DMWRITERBLOCK_WRITE_FAILED [] dmwriterblock.cpp:78
      [6] ERR_DMWRITERENCRYPT_WRITESECTOR_FAILED [] dmwriterencrypt.cpp:115
       [7] ERR_DMWRITERHTTP_WRITE_FAILED [] dmwriterhttp.cpp:35
        [8] ERR_DMHTTPMESSAGEOUTBOUND_WRITECONTENTPARTIAL_FAILED [] dmhttpmessageoutbound.cpp:1005
         [9] ERR_DMHTTPMESSAGEOUTBOUND_ENSUREHEADERSFLUSHED_FAILED [] dmhttpmessageoutbound.cpp:876
          [10] ERR_DMHTTPMESSAGEINBOUND_RECEIVE_FAILED [] dmhttpmessageinbound.cpp:797
           [11] ERR_DMCHANNELBUFFERED_READLINEWITHTIMEOUT_FAILED [] dmchannelsocket.cpp:196
            [12] ERR_DMCHANNELSOCKET_READUNBUFFERED_FAILED [ERR_ADD_TIMEOUT] dmchannelsocket.cpp:202

#4

If I’m reading this right, it looks like an HTTP connection was made your cloudian for an ujpload, some headers were sent, and then the moonwalk softweare got stuck reading data back. This sounds like it was doing an expect-100. IIRC, cloudian requires you to provide your own load-balancer. It could be that your LB doesn’t actually support the expect-100 stuff correctly. HTTP/1.1 servers are suppose to always support this though (see rfc2616/8.2.3) so you may need to just get a compliant load-balancer.