Issue 8788 - googleappengine - Every day around 9.00 AM Brussels time, huge drop in GAE performances - Google App Engine - Google Project Hosting

Feb 11, 2013

This is happening again!!!
STOP WHATHEVER YOU ARE DOING EVERY DAY AT THAT TIME!!!
 2013-02-11 09:22:21.721 /cron/StayUp 500 620652ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 I 2013-02-11 09:14:26.686 javax.servlet.ServletContext log: Initializing Spring root WebApplicationContext
 W 2013-02-11 09:19:59.527 com.csa.server.dao.DAO <clinit>: alert
 W 2013-02-11 09:22:21.368 Failed startup of context com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@144752d{/,/base/data/home/apps/myagendapro/20130210.36521548
 W 2013-02-11 09:22:21.368 <continued from previous message> at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458) ... 1 more
 W 2013-02-11 09:22:21.371 Nested in org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'verifier' defined in ServletContext resource [/WEB-I
 C 2013-02-11 09:22:21.387 Uncaught exception from servlet javax.servlet.UnavailableException: Initialization failed. at com.google.apphosting.runtime.jetty.AppVersionHandlerMa
 I 2013-02-11 09:22:21.651 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This requ
 I 2013-02-11 09:22:21.651 Process terminated because the request deadline was exceeded. (Error code 123)
 2013-02-11 09:22:17.753 /cron/StayUp 500 16868ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 I 2013-02-11 09:22:17.753 Request was aborted after waiting too long to attempt to service your request.
 2013-02-11 09:21:17.625 /cron/StayUp 500 17489ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 I 2013-02-11 09:21:17.625 Request was aborted after waiting too long to attempt to service your request.
 2013-02-11 09:20:17.064 /cron/StayUp 500 16959ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 I 2013-02-11 09:20:17.064 Request was aborted after waiting too long to attempt to service your request.
 2013-02-11 09:19:02.000 /cron/StayUp 200 1898ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 09:18:01.077 /cron/StayUp 200 467ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 09:17:18.444 /cron/StayUp 500 17307ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 I 2013-02-11 09:17:18.444 Request was aborted after waiting too long to attempt to service your request.
 2013-02-11 09:16:29.761 /cron/StayUp 500 16590ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 I 2013-02-11 09:16:29.761 Request was aborted after waiting too long to attempt to service your request.
 2013-02-11 09:11:21.201 /cron/StayUp 500 20490ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 W 2013-02-11 09:11:21.201 A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the nex
 2013-02-11 09:10:17.482 /cron/StayUp 500 16616ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 W 2013-02-11 09:10:17.481 A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the nex
 2013-02-11 09:09:19.594 /cron/StayUp 500 18820ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 I 2013-02-11 09:09:19.594 Request was aborted after waiting too long to attempt to service your request.
 2013-02-11 09:08:17.572 /cron/StayUp 500 17435ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 I 2013-02-11 09:08:17.572 Request was aborted after waiting too long to attempt to service your request.
 2013-02-11 09:07:07.690 /cron/StayUp 500 6811ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 W 2013-02-11 09:07:07.690 A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the nex
 2013-02-11 09:06:07.349 /cron/StayUp 200 3315ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 09:05:43.658 /cron/StayUp 500 103224ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 I 2013-02-11 09:05:43.658 Process terminated because the request deadline was exceeded. (Error code 123)
 2013-02-11 09:03:18.132 /cron/StayUp 500 17772ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 I 2013-02-11 09:03:18.132 Request was aborted after waiting too long to attempt to service your request.
 2013-02-11 09:02:14.678 /cron/StayUp 500 13962ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 W 2013-02-11 09:02:14.678 A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the nex
 2013-02-11 09:01:06.372 /cron/StayUp 200 6122ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 09:00:08.077 /cron/StayUp 200 6970ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:59:07.810 /cron/StayUp 200 7090ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:58:22.404 /cron/StayUp 200 4729ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:57:57.549 /cron/StayUp 200 417059ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 I 2013-02-11 08:52:32.196 javax.servlet.ServletContext log: Initializing Spring root WebApplicationContext
 2013-02-11 08:50:01.747 /cron/StayUp 200 1293ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:49:06.198 /cron/StayUp 200 5275ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:48:06.790 /cron/StayUp 200 6389ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:47:01.289 /cron/StayUp 200 326ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:46:01.906 /cron/StayUp 200 937ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:45:01.949 /cron/StayUp 200 1097ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:44:00.754 /cron/StayUp 200 328ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:43:00.325 /cron/StayUp 200 20ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:42:04.123 /cron/StayUp 200 3443ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:41:00.563 /cron/StayUp 200 389ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:40:01.087 /cron/StayUp 200 172ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:39:00.848 /cron/StayUp 200 28ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:38:00.777 /cron/StayUp 200 180ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:37:00.483 /cron/StayUp 200 268ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:36:00.144 /cron/StayUp 200 35ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:35:00.966 /cron/StayUp 200 217ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:34:00.662 /cron/StayUp 200 13ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:33:00.630 /cron/StayUp 200 231ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:32:01.316 /cron/StayUp 200 457ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:31:00.731 /cron/StayUp 200 16ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:30:00.678 /cron/StayUp 200 56ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:29:00.444 /cron/StayUp 200 123ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:28:00.275 /cron/StayUp 200 22ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:27:01.138 /cron/StayUp 200 425ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:26:00.665 /cron/StayUp 200 162ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:25:01.245 /cron/StayUp 200 208ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:24:00.713 /cron/StayUp 200 110ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
 2013-02-11 08:23:01.455 /cron/StayUp 200 588ms 0kb AppEngine-Google; (+http://code.google.com/appengine) 

Feb 11, 2013

What do we have to do to get your attention????

Feb 15, 2013

This morning 9.00 am brussels time it happens again!!
Instances are started but are not healthy. They cannot process any request and they cannot be shut down!!!
Beside them are 2 healthy instances which are fro any obscure reason not receiving any request.
I don't know what you are trying to achieve but we would be pleased to know it.
Until you get them: Could you implement temporary work around: to serve request only with healthy (instances that do respond to request) and warmed up instances)?
I'm in the process of quitting GAE and every day, you prove I'm right but until I do I hope that you'll solve this issue!

Feb 18, 2013

And AGAIN today same time same symptoms

Feb 18, 2013

How long does it take you to react on a Critical issue?
it's been opened for 10 days (occuring almost every week day since then and no comment from Google!

Feb 18, 2013

Come on please, this is getting beyond ridiculous. Not even any comment??
This represents the time of day my customers start getting busy in the system and need to be running fast, not a system you can't even log in to. I will lose my customers if this continues (after 2.5 yrs) and I will lose this business, and you will lose mine.
appid 'mybikeshopmanager'
2013-02-18 09:16:29.558
Failed startup of context com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@8d0b0f{/,/base/data/home/apps/mybikeshopmanager/248.365281273837482826}
org.mortbay.util.MultiException[java.lang.ExceptionInInitializerError, com.google.apphosting.runtime.HardDeadlineExceededError: This request (4583dd372ab484f4) started at 2013/02/18 08:15:22.608 UTC and was still executing at 2013/02/18 08:16:22.717 UTC.]
 at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:656)
 at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
 at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
 at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
 at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:219)
 at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194)
 at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
 at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447)
 at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
 at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
 at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
 at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
 at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
 at java.lang.Thread.run(Thread.java:679)
java.lang.ExceptionInInitializerError
 at com.tasteprovence.VVbooking.server.BookingServiceImpl.init(BookingServiceImpl.java:219)
 at javax.servlet.GenericServlet.init(GenericServlet.java:212)
 at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
 at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
 at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
 at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
 at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
 at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:219)
 at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194)
 at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
 at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447)
 at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
 at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
 at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
 at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
 at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
 at java.lang.Thread.run(Thread.java:679)
Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
 at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
 at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
 at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:1086)
 at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:914)
 at com.tasteprovence.VVbooking.server.util.PMF.<clinit>(PMF.java:8)
 ... 23 more
Caused by: java.lang.reflect.InvocationTargetException
 at com.google.appengine.runtime.Request.process-4583dd372ab484f4(Request.java)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at java.security.AccessController.doPrivileged(AccessController.java:63)
 at javax.jdo.JDOHelper.invoke(JDOHelper.java:1951)
 at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
 at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
 at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:1086)
 at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:914)
 at com.tasteprovence.VVbooking.server.util.PMF.<clinit>(PMF.java:8)
 at com.tasteprovence.VVbooking.server.BookingServiceImpl.init(BookingServiceImpl.java:219)
 at javax.servlet.GenericServlet.init(GenericServlet.java:212)
 at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
 at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
 at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
 at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
 at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
 at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
 at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
 at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
 at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
 ... 1 more
Caused by: com.google.apphosting.api.DeadlineExceededException: This request (4583dd372ab484f4) started at 2013/02/18 08:15:22.608 UTC and was still executing at 2013/02/18 08:16:22.052 UTC.
 at com.google.appengine.runtime.Request.process-4583dd372ab484f4(Request.java)
 at java.util.zip.ZipFile.read(Native Method)
 at java.util.zip.ZipFile.access$1200(ZipFile.java:57)
 at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:476)
 at java.util.zip.ZipFile$1.fill(ZipFile.java:259)
 at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
 at sun.misc.Resource.getBytes(Resource.java:124)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:273)
 at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
 at org.datanucleus.plugin.PluginParser.parseManifest(PluginParser.java:66)
 at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:415)
 at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:340)<continued in next message>
W 2013-02-18 09:16:29.558<continued from previous message>
 at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:222)
 at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:153)
 at org.datanucleus.plugin.PluginManager.registerExtensionPoints(PluginManager.java:82)
 at org.datanucleus.OMFContext.<init>(OMFContext.java:160)
 at org.datanucleus.OMFContext.<init>(OMFContext.java:141)
 at org.datanucleus.ObjectManagerFactoryImpl.initialiseOMFContext(ObjectManagerFactoryImpl.java:144)
 at org.datanucleus.jdo.JDOPersistenceManagerFactory.initialiseProperties(JDOPersistenceManagerFactory.java:316)
 at org.datanucleus.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:260)
 at org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory.<init>(DatastoreJDOPersistenceManagerFactory.java:71)
 at org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory.getPersistenceManagerFactory(DatastoreJDOPersistenceManagerFactory.java:126)
 ... 29 more
com.google.apphosting.runtime.HardDeadlineExceededError: This request (4583dd372ab484f4) started at 2013/02/18 08:15:22.608 UTC and was still executing at 2013/02/18 08:16:22.717 UTC.
 at com.google.appengine.runtime.Request.process-4583dd372ab484f4(Request.java)
 at java.io.FileInputStream.readBytes(Native Method)
 at java.io.FileInputStream.read(FileInputStream.java:231)
 at sun.misc.Resource.getBytes(Resource.java:124)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:273)
 at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
 at org.mortbay.util.Loader.loadClass(Loader.java:91)
 at org.mortbay.util.Loader.loadClass(Loader.java:71)
 at org.mortbay.jetty.servlet.Holder.doStart(Holder.java:73)
 at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:242)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
 at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
 at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
 at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
 at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
 at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
 at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
 at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
 at java.lang.Thread.run(Thread.java:679)
C 2013-02-18 09:16:29.564
Uncaught exception from servlet
javax.servlet.UnavailableException: Initialization failed.
 at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:228)
 at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194)
 at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
 at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447)
 at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
 at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
 at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
 at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
 at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
 at java.lang.Thread.run(Thread.java:679)

Feb 19, 2013

Is there anyone experiencing this issue that isn't on the M/S datastore?

Feb 21 (6 days ago)

same problem again this morning. Not doing any heavy processing, just logging in.
appid 'mybikeshopmanager'
2013-02-21 09:23:11.358
Error for /vvbooking/login
com.google.apphosting.runtime.HardDeadlineExceededError: This request (071c0b1df8a1e1a4) started at 2013/02/21 08:22:10.447 UTC and was still executing at 2013/02/21 08:23:10.837 UTC.
 at com.google.appengine.runtime.Request.process-071c0b1df8a1e1a4(Request.java)
 at java.util.zip.ZipFile.read(Native Method)
 at java.util.zip.ZipFile.access$1200(ZipFile.java:57)
 at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:476)
 at java.util.zip.ZipFile$1.fill(ZipFile.java:259)
 at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
 at sun.misc.Resource.getBytes(Resource.java:124)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:273)
 at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
 at org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:251)
 at org.datanucleus.jdo.JDOPersistenceManager.newQuery(JDOPersistenceManager.java:1298)
 at org.datanucleus.jdo.JDOPersistenceManager.newQuery(JDOPersistenceManager.java:1198)
 at org.datanucleus.jdo.JDOPersistenceManager.newQuery(JDOPersistenceManager.java:1352)
 at com.tasteprovence.VVbooking.server.core.LoginHelper.setRoleAndMappings(LoginHelper.java:149)
 at com.tasteprovence.VVbooking.server.core.LoginHelper.getShopMapping(LoginHelper.java:94)
 at com.tasteprovence.VVbooking.server.LoginServiceImpl.login(LoginServiceImpl.java:114)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:43)
 at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:569)
 at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
 at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
 at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
 at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
 at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
 at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
 at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
 at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.Server.handle(Server.java:326)
 at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
 at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
 at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
 at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
 at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
 at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
 at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
 at java.lang.Thread.run(Thread.java:679)
C 2013-02-21 09:23:11.366
Uncaught exception from servlet
com.google.apphosting.runtime.HardDeadlineExceededError: This request (071c0b1df8a1e1a4) started at 2013/02/21 08:22:10.447 UTC and was still executing at 2013/02/21 08:23:10.837 UTC.
 at com.google.appengine.runtime.Request.process-071c0b1df8a1e1a4(Request.java)
 at java.util.zip.ZipFile.read(Native Method)
 at java.util.zip.ZipFile.access$1200(ZipFile.java:57)
 at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:476)
 at java.util.zip.ZipFile$1.fill(ZipFile.java:259)
 at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
 at sun.misc.Resource.getBytes(Resource.java:124)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:273)
 at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
 at org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:251)
 at org.datanucleus.jdo.JDOPersistenceManager.newQuery(JDOPersistenceManager.java:1298)
 at org.datanucleus.jdo.JDOPersistenceManager.newQuery(JDOPersistenceManager.java:1198)
 at org.datanucleus.jdo.JDOPersistenceManager.newQuery(JDOPersistenceManager.java:1352)
 at com.tasteprovence.VVbooking.server.core.LoginHelper.setRoleAndMappings(LoginHelper.java:149)
 at com.tasteprovence.VVbooking.server.core.LoginHelper.getShopMapping(LoginHelper.java:94)
 at com.tasteprovence.VVbooking.server.LoginServiceImpl.login(LoginServiceImpl.java:114)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:43)
 at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:569)
 at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
 at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
 at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
 at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
 at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
 at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
 at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
 at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.Server.handle(Server.java:326)
 at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
 at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
 at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
 at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
 at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
 at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
 at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
 at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
 at java.lang.Thread.run(Thread.java:679)
I 2013-02-21 09:23:11.541
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
W 2013-02-21 09:23:11.541
A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)

Feb 21 (5 days ago)

Since no one answered my question in the affirmative, I'm going to assume everyone experiencing this issue is using M/S. Upgrading to HRD will solve your issue. M/S is deprecated and there is a clear and straightforward path to migrating to HRD.

Feb 21 (5 days ago)

Hi,
I've suffered non trivial slowdowns in the same timeframe: -
- The app was apparently temporarily frozen.
- Browsers just showed a "loading" spinner.
- Console showed no instances.
This went on for around 15 mins.
Differences between my app and the OPs are: -
- Mine is HRD (vs. M/S)
- Mine is Python (vs. Java)
- I'm in the UK, and the problem was at 8am our time (as opposed to 9am in Belgium).
Given how much trouble the OP has had with this issue, I think that your assumption is not warranted by the circumstances, and is in any event premature.
--
Cheers,
PhilK

Yesterday (31 hours ago)

AND TODAY AGAIN!!!!!!!!!
GAE is just giving 500 errors to all requests!!! Application completely down in peak hour!!!
THANKS A LOT FOR YOUR SUPPORT.
What do we need to get some help? Contact the press?
HELP!!!!!!!!!!!
If you read this post and know some people at Google. Could you ask them what's happening?

Yesterday (31 hours ago)

I am facing same error.
app id : "buddy-rush"
I am using followings.
 - python2.5
 - Django
 - master / slave

Yesterday (31 hours ago)

Please look at #12: this happens on HRD as well. Beside this the issue seems to be linked to instance initialization

Today (6 hours ago)

From https://code.google.com/p/googleappengine/issues/detail?id=8881 :
This affects all our non-HRD instances (for instance androidpickup, appbrainz and promotedapps)
Since a number of days there is a ~20 minute outage of datastore *every single day* at 9AM CET / midnight PDT. The error count spikes up and average milliseconds / request goes into the 30-40s / request. Our sites are indistinguishable from being down altogether during this timeframe.
The appengine status dashboard doesn't show anything out of the ordinary (but we've learned from the past that this doesn't have to mean anything, and the bug to make the status page actually reflect the status of the systems has been open for over 3 years now: https://code.google.com/p/googleappengine/issues/detail?id=2377 ).
We are in the process of finding a solution to migrating off our non-HRD instances, is it possible to fix these big outages happening every day?

Today (4 hours ago)

I got here from HackerNews, but after seeing the original poster spam the forums in multiple places and have a bad attitude, I can't blame Google for not fixing what looks to me like a non-issue.
Fuck 'em.

Today (4 hours ago)

@ lysprr@gmail.com - Very constructive. 
As for bad attitude, losing credibility with your clients while your supplier ignores you will do that. I consider myself lucky that I've only experienced this as extreme slowdown / no instances, and that the time slot in question is outside my clients working hours here in the UK (although not mine).

Today (2 hours ago)

@ lysprr@gmail.com, why is this a "non-issue"?