Refactor misleading database errors

[tanyongzhi]

This PR fixes misleading database errors. In particular, this PR, when applied, will:

- Fix mongo database initialization hanging during local development if local mongo instance is not started

• In development, the timeout for database connection is 0 - corresponding to a blocking call to DialWithInfo(). This effectively leads to the service being unresponsive, as the services will forever be waiting on the connection to the database during initialization without logging anything meaningful. As a result, all subsequent requests to the service will simply be timed out and return 500 errors.
• This is fixed by setting the timeout for database connections to 60 both in production and in development. Correspondingly, DialWithInfo() will return an error, which will be logged. In the future, we can consider exiting the service upon initialization failure.

- Change not found errors to return 404 instead of 500

• Previously, many GET requests that retrieve data from mongo will return a 500 error if the data is not present in the database. This is misleading, and 500 errors should be reserved for internal connection errors with the database backend. This is fixed by changing the database error to a 404 status code if the database returns a not found error.

[lauzadis]

I tested this locally and it didn't seem to work. If the API can't make a connection to mongodb within 60 seconds, it should crash?

[tanyongzhi]

Are you running all services with make run? If so, can you check which services are still running after the mongo timeout happens? The gateway and the stats service should still be running, as these services do not access mongo and would not be terminated on connection failure. All other services should be terminated.

While on this topic, perhaps we should change run.sh to terminate all services if anyone exits with an error...

[lauzadis]

Ah no, I tested it with make setup which (ideally) should crash immediately once it detects mongod is not running. I didn't realize this fix was for make run.

It still looks like those services aren't dying after missing the mongod connection:

matas      35633   35628  3 16:25 pts/1    00:00:09 /home/matas/Desktop/api/bin/hackillinois-api --service user
matas      35634   35628  3 16:25 pts/1    00:00:09 /home/matas/Desktop/api/bin/hackillinois-api --service registration
matas      35635   35628  3 16:25 pts/1    00:00:09 /home/matas/Desktop/api/bin/hackillinois-api --service decision
matas      35636   35628  3 16:25 pts/1    00:00:08 /home/matas/Desktop/api/bin/hackillinois-api --service rsvp
matas      35637   35628  3 16:25 pts/1    00:00:09 /home/matas/Desktop/api/bin/hackillinois-api --service checkin
matas      35638   35628  3 16:25 pts/1    00:00:09 /home/matas/Desktop/api/bin/hackillinois-api --service upload
matas      35639   35628  3 16:25 pts/1    00:00:09 /home/matas/Desktop/api/bin/hackillinois-api --service mail
matas      35640   35628  3 16:25 pts/1    00:00:09 /home/matas/Desktop/api/bin/hackillinois-api --service event
matas      35642   35628  0 16:25 pts/1    00:00:00 /home/matas/Desktop/api/bin/hackillinois-api --service stat
matas      35647   35628  3 16:25 pts/1    00:00:09 /home/matas/Desktop/api/bin/hackillinois-api --service notifications
matas      35651   35628  3 16:25 pts/1    00:00:09 /home/matas/Desktop/api/bin/hackillinois-api --service project
matas      35656   35628  3 16:25 pts/1    00:00:09 /home/matas/Desktop/api/bin/hackillinois-api --service profile
matas      35661   35628  3 16:25 pts/1    00:00:08 /home/matas/Desktop/api/bin/hackillinois-api --service room
matas      35666   35628  0 16:25 pts/1    00:00:00 /home/matas/Desktop/api/bin/hackillinois-api --service gateway

(ignore the missing auth service, I killed it manually)

Let's go over this in tomorrow's meeting, maybe I am missing something simple

[tanyongzhi]

Sorry for the confusion! But yes, let's go over this tomorrow.

[TakshPSingh]

@tanyongzhi do you think it might be a better idea to move the 404 error part into a separate commit so that we can merge it now before the rest of the commit is approved?