이슈
가끔씩 서버가 접속이 안됩니닷. 😫
$ ssh -i ~/devcourse/project/jungdam/ec2-jungdam.cer ubuntu@52.78.146.201 ssh: connect to host 52.78.146.201 port 22: Operation timed out
EC2 로 확인 해보니 인스턴스 실패가 나오네요
원인 분석
Code Deploy가 배포하면서 남기는 로그를 살펴보겠습니다.
2021-12-16 01:38:14 INFO [codedeploy-agent(757)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb. 2021-12-16 01:38:59 INFO [codedeploy-agent(757)]: [Aws::CodeDeployCommand::Client 200 45.640318 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:ap-northeast-2:296441481802:instance/i-021afdef41f2dc755") 2021-12-16 01:39:00 INFO [codedeploy-agent(757)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb. 2021-12-16 01:45:25 INFO [codedeploy-agent(757)]: [Aws::CodeDeployCommand::Client 200 290.310856 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:ap-northeast-2:296441481802:instance/i-021afdef41f2dc755") 2021-12-16 01:53:34 INFO [codedeploy-agent(757)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb. 2021-12-16 02:23:41 INFO [codedeploy-agent(757)]: [Aws::CodeDeployCommand::Client 0 1699.055561 3 retries] poll_host_command(host_identifier:"arn:aws:ec2:ap-northeast-2:296441481802:instance/i-021afdef41f2dc755") Seahorse::Client::NetworkingError end of file reached 2021-12-16 02:23:42 ERROR [codedeploy-agent(757)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Network error: #<Seahorse::Client::NetworkingError: end of file reached> 2021-12-16 02:24:28 INFO [codedeploy-agent(757)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb. 2021-12-16 02:27:58 INFO [codedeploy-agent(757)]: [Aws::CodeDeployCommand::Client 200 200.022736 1 retries] poll_host_command(host_identifier:"arn:aws:ec2:ap-northeast-2:296441481802:instance/i-021afdef41f2dc755") 2021-12-16 02:27:59 INFO [codedeploy-agent(757)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb. 2021-12-16 02:36:38 INFO [codedeploy-agent(757)]: [Aws::CodeDeployCommand::Client 200 342.649119 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:ap-northeast-2:296441481802:instance/i-021afdef41f2dc755") 2021-12-16 02:45:44 INFO [codedeploy-agent(757)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb. 2021-12-16 03:42:10 INFO [codedeploy-agent(757)]: [Aws::CodeDeployCommand::Client 0 3245.862599 3 retries] poll_host_command(host_identifier:"arn:aws:ec2:ap-northeast-2:296441481802:instance/i-021afdef41f2dc755") Seahorse::Client::NetworkingError Failed to open TCP connection to codedeploy-commands.ap-northeast-2.amazonaws.com:443 (getaddrinfo: Temporary failure in name resolution) 2021-12-16 03:42:10 ERROR [codedeploy-agent(757)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Network error: #<Seahorse::Client::NetworkingError: Failed to open TCP connection to codedeploy-commands.ap-northeast-2.amazonaws.com:443 (getaddrinfo: Temporary failure in name resolution)> 2021-12-16 03:42:11 INFO [codedeploy-agent(757)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb. 2021-12-16 03:42:13 INFO [codedeploy-agent(757)]: [Aws::CodeDeployCommand::Client 0 2.110911 3 retries] poll_host_command(host_identifier:"arn:aws:ec2:ap-northeast-2:296441481802:instance/i-021afdef41f2dc755") Seahorse::Client::NetworkingError Failed to open TCP connection to codedeploy-commands.ap-northeast-2.amazonaws.com:443 (getaddrinfo: Temporary failure in name resolution) 2021-12-16 03:42:13 ERROR [codedeploy-agent(757)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Network error: #<Seahorse::Client::NetworkingError: Failed to open TCP connection to codedeploy-commands.ap-northeast-2.amazonaws.com:443 (getaddrinfo: Temporary failure in name resolution)> 2021-12-16 03:42:24 INFO [codedeploy-agent(757)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb. 2021-12-16 03:42:26 INFO [codedeploy-agent(757)]: [Aws::CodeDeployCommand::Client 0 2.116644 3 retries] poll_host_command(host_identifier:"arn:aws:ec2:ap-northeast-2:296441481802:instance/i-021afdef41f2dc755") Seahorse::Client::NetworkingError Failed to open TCP connection to codedeploy-commands.ap-northeast-2.amazonaws.com:443 (getaddrinfo: Temporary failure in name resolution)
살펴보면 11시 (02시는 UTC 기준입니다) 부터 뭔가 fail이 나오네요 왜!! 대체 왜!! (제 생각에 메모리 부족인듯)
원인을 찾을 수가 없네요... 고도화 작업때 ec2 메모리 모니터링도 넣어봐야겠어요.
해결 방법
- 계정 로그인 후 인스턴스 재부팅을 진행하였습니다. 또한 이러한 사항이 발생한다면 watch와 인스턴스 재부팅 로직을 넣었습니다. (돈드나,..?)
- 또한 ec2 인스턴스가 reboot 될 때 Docker container 또한 자동으로 켜져야 하기 때문에,
docker run --restart=unless-stopped옵션을 넣어 주었습니다.
추가
- 서버인스턴스상태체크가 실패인 경우 경보를 생성하여 제 메일로 보내는 로직을 추가합니다.
