Hibernate MultipleHiLoPerTableGenerator deadlock and TableGenerator.allocationSize

The other week I had the pleasure of visiting a former client, which had a problem with the connection pool being completely drained for connections, and as a consequence of that, their system pretty much locking up.

A dump of the stacks of all threads when the system locked up, told us the culprit was a specific web-service call, and all the stacks were dead in a call to hibernates MultipleHiLoPerTableGenerator.generate method. Like this:

38018:http-7555-140, state=BLOCKED
	org.hibernate.id.MultipleHiLoPerTableGenerator.generate(MultipleHiLoPerTableGenerator.java:204)
	org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:122)
	org.hibernate.ejb.event.EJB3PersistEventListener.saveWithGeneratedId(EJB3PersistEventListener.java:69)
	org.hibernate.event.def.DefaultPersistEventListener.entityIsTransient(DefaultPersistEventListener.java:179)
	org.hibernate.event.def.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:135)
	org.hibernate.event.def.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:61)
	org.hibernate.impl.SessionImpl.firePersist(SessionImpl.java:800)
	org.hibernate.impl.SessionImpl.persist(SessionImpl.java:774)
	org.hibernate.impl.SessionImpl.persist(SessionImpl.java:778)
	org.hibernate.ejb.AbstractEntityManagerImpl.persist(AbstractEntityManagerImpl.java:668)
	sun.reflect.GeneratedMethodAccessor553.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	java.lang.reflect.Method.invoke(Method.java:597)
	org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:365)
	$Proxy62.persist(Unknown Source)
	sun.reflect.GeneratedMethodAccessor553.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	java.lang.reflect.Method.invoke(Method.java:597)
	org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:240)
	$Proxy62.persist(Unknown Source)
	org.springframework.orm.jpa.JpaTemplate$5.doInJpa(JpaTemplate.java:264)
	org.springframework.orm.jpa.JpaTemplate.execute(JpaTemplate.java:183)
	org.springframework.orm.jpa.JpaTemplate.persist(JpaTemplate.java:262)

What’s interesting of this particular web-service call is, that it generates a lot of new rows of a specific JPA mapped entity in the system. Hence, it calls the id-generator a lot of times. And when the system is under heavy load, this particular web-service method is called often, which generates lots of parallel transactions, all trying to generate a lot of ids.

The class was mapped like this:

@Entity
public class DetailRow implements Serializable {

  private Long id;

  @Id
  @Column(unique = true, nullable = false)
  @GeneratedValue(strategy = GenerationType.TABLE, generator = "DetailRow.id")
  @TableGenerator(name = "DetailRow.id", table = "IdSequence", pkColumnName = "Name", valueColumnName = "WaterMark", pkColumnValue = "DetailRow", allocationSize = 1)
  public Long getId() {
    return id;
  }

  public void setId(Long id) {
    this.id = id;
  }
}

Now, it turns out others have had this problem too. The problem is with the MultipleHiLoPerTableGenerator of hibernate, which allocates new ids in a separate transaction (which it must, so that’s kinda okay). But, because it grabs an extra connection while holding an existing connection, it can deadlock when other calls do the same at the same time. The calls can end up hold the one connection all the others need. This is no news, and people have filed bugs (HB-1246 and HHH-1739) in the hibernate bug-db for it. None of them with a fix though.

I myself have no final fix for it. But I have an easy solution, which, at least for us, made the probability for the deadlock so small, that it stopped happening.

Solution

The use of @TableGenerator with hibernate as ORM provider, maps underneath to MultipleHiLoPerTableGenerator. I think we got that covered by now :-) But, there is an interesting parameter “allocationSize” to @TableGenerator. This makes the generator work in “chunks of ids”. Given the “IdSequence” table have a watermark of “123″, and we have set allocationSize to “1000″, the generator will increment watermark to “124″ ONCE and then for the next 1000 ids allocate 123000, 123001, 123002, ..123999 and first THEN, will it touch the “IdSequence” table again, to increment the watermark.

Setting “allocationSize” on @TableGenerator severely lowers the number of times an extra connection is needed, hence also making it a lot less probable, that it will end up in a deadlock. The “bug” is still there, because, technically, it still can happen. But practically, it won’t happen, if setting “allocationSize” high enough.

Caveat

Not many.

Remember that “allocationSize” is multiplied onto the “watermark” value, so to avoid a large unused hole of ids when deploying a new version with a larger “allocationSize”, you should divide the watermark with “allocationSize”.

Also, there can (will) be holes in the sequence, as unused ids from a “chunk” are lost when process dies. No big deal for us. Might be for some.

And what is a “high enough” value for “allocationSize” then? Well, it depends on your system. How many rows are created, and at which rate, and with how many concurrent transactions, related to your connection pool size. Do the math for loaded scenarios and try setting it to what you think is “high enough”.

August 15, 2011 В· polesen В· No Comments
Tags: , ,  В· Posted in: Programming

Leave a Reply