Xcode 4.2 + iOS5 building armv6+armv7 binary problems

After upgrading to Xcode4.2 and iOS5, we had problems building the binary for submission to AppStore. The final binary did not include both armv6 and armv7 architectures, so it was rejected. When linking, the linker told us about this warning:

warning: iPhone/iPod Touch: application executable is missing a required architecture.
At least one of the following architecture(s) must be present: armv6 (-19033)

I think the use of this Xcode setting “$(ARCHS_STANDARD_32_BIT)” has changed its meaning since,..sometime. We solved this by doing this:

  • Goto Project->Build Settings
  • Find Architectures and change it from $(ARCHS_STANDARD_32_BIT) to two new values armv6 and armv7
  • While you are there, you can also check that Build Active Architecture Only has been set to No for the AdHoc and AppStore targets

October 30, 2011 · polesen · 3 Comments
Tags: ,  Â· Posted in: Tools

iOS Gotcha: Empty NSArray Optimization

I got bitten by a little iOS optimization the other day. It turns out, that NSArray has been optimized to represent the empty array with the same singleton object, no matter how instantiated.

This code:

NSArray *a1 = [[NSArray alloc] init];
NSArray *a2 = [[NSArray alloc] init];
NSLog(@"a1 == a2 => %d", (a1 == a2));

says:

a1 == a2 => 1

which indicates, that even though it looks like we instantiate 2 objects, the “a1″ and “a2″ variables point to the same object.

It doesn’t matter much how we instantiate the empty NSArray. We could also do:

NSArray *emptyArr = [[NSArray alloc] init];
NSArray *a1 = [NSArray arrayWithArray:emptyArr];
NSArray *a2 = [NSArray arrayWithArray:emptyArr];

// or

NSArray *a1 = [NSArray array];
NSArray *a2 = [NSArray array];

// or

NSArray *a1 = [NSArray arrayWithObjects:nil];
NSArray *a2 = [NSArray arrayWithObjects:nil];

I guess it is because NSArray is immutable, that this optimization was found okay to make. If I add just one object, I can’t reproduce it. So this:

NSArray *a1 = [NSArray arrayWithObject:@"foo"];
NSArray *a2 = [NSArray arrayWithObject:@"foo"];
NSLog(@"a1 == a2 => %d", (a1 == a2));

yields:

a1 == a2 => 0

and maybe more surprising, this code:

NSArray *a1 = [NSArray arrayWithObject:@"foo"];
NSArray *a2 = [NSArray arrayWithArray:a1];
NSLog(@"a1 == a2 => %d", (a1 == a2));

also yields:

a1 == a2 => 0

I would’ve guessed, that “[NSArray arrayWithArray:a1]” simply returned “a1″, because after all, NSArray is immutable. But it seems not to, when the given array isn’t an empty array.

Anyways, I got bitten because I had some code in a setter method with an optimization that went something like this:

if (_ivarArray != newArray) {
   // work the magic
}

and I really wanted to “work the magic”, even though “_ivarArray” previously also was an empty array.

October 28, 2011 · polesen · No Comments
Tags:  Â· Posted in: Programming

Dead Simple dos2unix “command”

The web is floating over (well, .. nearly) with various one-liners that can convert newlines of a text file from the dos/windows way to the unix way. These use tools like sed, tr, perl, .. to do the trick, and here is another one – using vim:

$ vi your-dos-newline-file.txt
:set fileformat=unix
:wq
and you are done. And I am pretty sure this can be one-lined too, with the proper v(h)im-sical command tooling :-)

October 10, 2011 · polesen · No Comments
Posted in: Operating Systems, Tools

Universal Library with both Simulator and iOS Architectures

Just a quick note to myself, because today I had to dig out from my brain, how it was I once built a Mach-O Universal library with both Simulator (i386) architecture and iOS (armv6+7) architectures in it.

xcodebuild -configuration Release -target my -sdk iphonesimulator4.0 clean build
xcodebuild -configuration Release -target my -sdk iphoneos4.3 clean build
lipo -create -output build/libmy.a build/Release-iphonesimulator/libmy.a build/Release-iphoneos/libmy.a

The final fat library created by “lipo” can now be used to link both Simulator and iOS builds.

When you construct the above lines for your project, a couple of xcodebuild commands that list info about a given xcode project, might come in handy:

xcodebuild -list yourProjectName
xcodebuild -showsdks

Where “-list” list information like “targets” and “build configurations” about given project and “-showsdks” shows the possible sdk names to use.

August 22, 2011 · polesen · No Comments
Tags: ,  Â· Posted in: Programming, Tools

Hibernate MultipleHiLoPerTableGenerator deadlock and TableGenerator.allocationSize

The other week I had the pleasure of visiting a former client, which had a problem with the connection pool being completely drained for connections, and as a consequence of that, their system pretty much locking up.

A dump of the stacks of all threads when the system locked up, told us the culprit was a specific web-service call, and all the stacks were dead in a call to hibernates MultipleHiLoPerTableGenerator.generate method. Like this:

38018:http-7555-140, state=BLOCKED
	org.hibernate.id.MultipleHiLoPerTableGenerator.generate(MultipleHiLoPerTableGenerator.java:204)
	org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:122)
	org.hibernate.ejb.event.EJB3PersistEventListener.saveWithGeneratedId(EJB3PersistEventListener.java:69)
	org.hibernate.event.def.DefaultPersistEventListener.entityIsTransient(DefaultPersistEventListener.java:179)
	org.hibernate.event.def.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:135)
	org.hibernate.event.def.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:61)
	org.hibernate.impl.SessionImpl.firePersist(SessionImpl.java:800)
	org.hibernate.impl.SessionImpl.persist(SessionImpl.java:774)
	org.hibernate.impl.SessionImpl.persist(SessionImpl.java:778)
	org.hibernate.ejb.AbstractEntityManagerImpl.persist(AbstractEntityManagerImpl.java:668)
	sun.reflect.GeneratedMethodAccessor553.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	java.lang.reflect.Method.invoke(Method.java:597)
	org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:365)
	$Proxy62.persist(Unknown Source)
	sun.reflect.GeneratedMethodAccessor553.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	java.lang.reflect.Method.invoke(Method.java:597)
	org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:240)
	$Proxy62.persist(Unknown Source)
	org.springframework.orm.jpa.JpaTemplate$5.doInJpa(JpaTemplate.java:264)
	org.springframework.orm.jpa.JpaTemplate.execute(JpaTemplate.java:183)
	org.springframework.orm.jpa.JpaTemplate.persist(JpaTemplate.java:262)

What’s interesting of this particular web-service call is, that it generates a lot of new rows of a specific JPA mapped entity in the system. Hence, it calls the id-generator a lot of times. And when the system is under heavy load, this particular web-service method is called often, which generates lots of parallel transactions, all trying to generate a lot of ids.

The class was mapped like this:

@Entity
public class DetailRow implements Serializable {

  private Long id;

  @Id
  @Column(unique = true, nullable = false)
  @GeneratedValue(strategy = GenerationType.TABLE, generator = "DetailRow.id")
  @TableGenerator(name = "DetailRow.id", table = "IdSequence", pkColumnName = "Name", valueColumnName = "WaterMark", pkColumnValue = "DetailRow", allocationSize = 1)
  public Long getId() {
    return id;
  }

  public void setId(Long id) {
    this.id = id;
  }
}

Now, it turns out others have had this problem too. The problem is with the MultipleHiLoPerTableGenerator of hibernate, which allocates new ids in a separate transaction (which it must, so that’s kinda okay). But, because it grabs an extra connection while holding an existing connection, it can deadlock when other calls do the same at the same time. The calls can end up hold the one connection all the others need. This is no news, and people have filed bugs (HB-1246 and HHH-1739) in the hibernate bug-db for it. None of them with a fix though.

I myself have no final fix for it. But I have an easy solution, which, at least for us, made the probability for the deadlock so small, that it stopped happening.

Solution

The use of @TableGenerator with hibernate as ORM provider, maps underneath to MultipleHiLoPerTableGenerator. I think we got that covered by now :-) But, there is an interesting parameter “allocationSize” to @TableGenerator. This makes the generator work in “chunks of ids”. Given the “IdSequence” table have a watermark of “123″, and we have set allocationSize to “1000″, the generator will increment watermark to “124″ ONCE and then for the next 1000 ids allocate 123000, 123001, 123002, ..123999 and first THEN, will it touch the “IdSequence” table again, to increment the watermark.

Setting “allocationSize” on @TableGenerator severely lowers the number of times an extra connection is needed, hence also making it a lot less probable, that it will end up in a deadlock. The “bug” is still there, because, technically, it still can happen. But practically, it won’t happen, if setting “allocationSize” high enough.

Caveat

Not many.

Remember that “allocationSize” is multiplied onto the “watermark” value, so to avoid a large unused hole of ids when deploying a new version with a larger “allocationSize”, you should divide the watermark with “allocationSize”.

Also, there can (will) be holes in the sequence, as unused ids from a “chunk” are lost when process dies. No big deal for us. Might be for some.

And what is a “high enough” value for “allocationSize” then? Well, it depends on your system. How many rows are created, and at which rate, and with how many concurrent transactions, related to your connection pool size. Do the math for loaded scenarios and try setting it to what you think is “high enough”.

August 15, 2011 · polesen · No Comments
Tags: , ,  Â· Posted in: Programming

Problems with require ‘mongo_mapper’ on cloudfoundry

I’ve been playing a bit with cloudfoundry today, and I wanted to do a simple example that connected to a mongodb service from ruby. Should be simple but I had some trouble. Some of the trouble I am sure stems from the fact that my ruby skills are somewhat rusty. Other from a somewhat alpha-like experience of the cloudfoundry product, which is supposed to be in beta :-)

I started out simple, with these ruby require lines in the top of my ruby sinatra app:

require 'sinatra'
require 'json'
require 'mongo'
require 'mongo_mapper'

Unfortunately, that made vmc update of my app fail. Cloudfoundry couldn’t start the app, and there were no logs or crashlogs to be seen :-( A little trial-n-error told me that it was the require of ‘mongo_mapper’, that made it fail. After some more trial-n-errors and a lot of googling, I could put together the needed bits.

This page at the cloudfoundry site has good info on what is required (and what they know won’t work) when running ruby apps on cloudfoundry. It also said, that I should be using Bundler and a Gemfile to describe my gem dependencies and package the app before deployment. So, I ended up with this Gemfile:

source "http://rubygems.org"
gem 'sinatra'
gem 'json'
gem 'mongo'
gem 'mongo_mapper'

and changed the require-lines in the sinatra application ruby file with this single line:

Bundler.require

When using bundler we then need to do these two steps before vmc push or update (each time dependencies have changed):

bundle package
bundle install

If you don’t have bundler, you can simply do:

sudo gem install bundler

June 10, 2011 · polesen · No Comments
Tags: , , ,  Â· Posted in: Programming

Springified Servlets, Listeners and Filters

Here are some short, but fully functional, templates for when you have a need for HttpServlet, ContextListener or Filter implementations, that also need spring bean dependencies.

You use them by subclassing them and calling the getBean method when a dependency is needed, like this:

// ..blah blah..inside subclass
MyService myService = getBean("myService");

Okay, here they come..

SpringHttpServlet

package net.techper.server.web;

import org.springframework.context.ApplicationContext;
import org.springframework.web.context.support.WebApplicationContextUtils;

import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;

public abstract class SpringHttpServlet extends HttpServlet {

  private ApplicationContext context;

  @Override
  public void init(ServletConfig config) throws ServletException {
    super.init(config);
    context = WebApplicationContextUtils.getRequiredWebApplicationContext(config.getServletContext());
  }

  @Override
  public void destroy() {
    context = null;
    super.destroy();
  }

  protected <T> T getBean(String beanName) {
    return (T) context.getBean(beanName);
  }
}

SpringContextListener

package net.techper.server.web;

import org.springframework.context.ApplicationContext;
import org.springframework.web.context.support.WebApplicationContextUtils;

import javax.servlet.ServletContext;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;

public abstract class SpringContextListener implements ServletContextListener {

  private ApplicationContext context;

  @Override
  public void contextInitialized(ServletContextEvent sce) {
    context = WebApplicationContextUtils.getRequiredWebApplicationContext(sce.getServletContext());
  }

  @Override
  public void contextDestroyed(ServletContextEvent sce) {
    context = null;
  }

  protected <T> T getBean(String beanName) {
    return (T) context.getBean(beanName);
  }
}

SpringFilter

package net.techper.server.web;

import org.springframework.context.ApplicationContext;
import org.springframework.web.context.support.WebApplicationContextUtils;

import javax.servlet.Filter;
import javax.servlet.FilterConfig;
import javax.servlet.ServletContext;
import javax.servlet.ServletException;

public abstract class SpringFilter implements Filter {

  private ApplicationContext context;

  @Override
  public void init(FilterConfig filterConfig) throws ServletException {
    context = WebApplicationContextUtils.getRequiredWebApplicationContext(filterConfig.getServletContext());
  }

  @Override
  public void destroy() {
    context = null;
  }

  protected <T> T getBean(String beanName) {
    return (T) context.getBean(beanName);
  }
}

March 5, 2011 · polesen · 2 Comments
Tags: , , , ,  Â· Posted in: Programming

Session Issues with Grails Command Objects

I’ve been doing a bit of Grails programming lately, and I must say it has been a nice experience so far. I really like the near instant turn-around time where I fix something in the controller or the view, and a refresh in the browser shows the new code running.

But…

I started using grails command objects (a bit like Struts form beans) to have the submitted form data data-bound and validated to an object different from the model. Like this:

class UploadDataSetController {
  def uploadDataSetFile = { UploadDataSetFileCommand cmd ->
    if (cmd.hasErrors()) {
      // foo
    } else {
      // bar
      session["UploadDataSetController.UploadDataSetFileCommand"] = cmd
    }
  }
}

class UploadDataSetFileCommand {
   // blah blah
}

As you can see above, I defined the command class inside the file of the controller, which is what the documentation recommends.

But, have a look at what I do in the else block. I put the command instance into session storage. And what is the problem with that? Well, when I change something (no matter what) in the controller file, Grails will recompile it on the fly (which is what I like). But, recompiling will make the class file of the command class different from the class of the instance in the session, giving me ClassCastException when reloading in the browser :-(

I tried moving the command class definition into its own file, which works fine. Only thing is, these won’t be recompiled at runtime, so I need to take the server down and up all the time, when I develop on the command class to get the changes. Not nice.

Actually, I tried something different. I tried keeping the command class definition in the file of the controller, but also providing it with a serialVersionUID definition. But it still failed on me. I think the reason for this being the classloader changing on recompile/refresh.

What I ended up with was the command class definition in the controller when developing heavily on it, and moving it into a separate file after it stabilized, which opened up for changes to the controller without problems.

Any good ideas for better solutions on this?

February 5, 2011 · polesen · One Comment
Tags:  Â· Posted in: Programming

“ls” Command Slow on Very Large Directories

I had some problems with listing files on linux from a directory with pretty many files in it (around 700.000 files now and still counting). Doing a simple ls on the directory never returned (or at least I killed it after a couple of hours waiting).

Now, actually I did know this could be a problem, because unix-like filesystems have had this scaling problem with very large directories for some time now. But I thought I was in the clear, as I was only doing a simple “ls”, which should be able to read all the filenames directly from the directory file. (As opposed to an “ls -l” which actually needs to stat() each inode to get to the extra information needed to present the long listing.)

Another thing that I thought could help me was the fact, that it was a gpfs filesystem, which should have support for very large directories.

The “solution” turned out to be simple. It all dawned on me when I was trying to kill the “ls” command that was hanging. I did a “ps” + “grep” after it, and got this command listing from “ps” output:

/bin/ls -N --color=tty -T 0 /my/large/dir

Ahh, an “alias”. On my system, “ls” was an alias to "/bin/ls $LS_OPTIONS", and LS_OPTIONS was set to -N --color=tty -T 0. I am not sure, but I suspect it being the "--color=tty" option, because to color files according to their types and attributes must incur a stat() on each inode.

And gpfs actually does seem to scale well with very large directories. I mean, maybe it is okay that it takes for ages to stat() 700.000+ inodes, but each single stat() call is pretty quick, if you know the exact filename in advance.

January 25, 2011 · polesen · 3 Comments
Tags: , ,  Â· Posted in: Operating Systems

Wine Slow to Start – Fontconfig Issue

Here is a solution to a problem we’ve had with wine that someone might find helpful, in case they experienced the same.

Part of a web application we’ve build is dispatching some calculation functionality to a windows based cmdline program. Because we are hosting the application on linux, we are using wine for execution of the calculation program. Suddenly one day, after applying a bunch of bugfix patches, wine became painfully slow to startup. We are talking 25 seconds or so, for what used to take around 1-2 seconds.

After a lot of digging and searching, we found that wine was using a lot of resources on reading fonts. Okay, but why? No new fonts had been installed. After some further digging, in turned out that rebuilding the fontconfig cache, made wine slow. Hugh! A cache of fonts should make it faster.

Yes, using a font cache will make font resolve much faster – if the cache is in a format, that the program using fontconfig understand. It turned out, that the SLES 10 we are running, is using a dead-old 2.3.x version of fontconfig. And this old version of fontconfig only has one set of cache files in “/var/cache/fontconfig“. But, fontconfig builds the cache differently for 32bit and 64bit architectures. Building with 64bit version of fc-cache overwrites the 32bit cache files and vice versa. Our SLES installation and machine architecture is 64bit, but wine is a 32bit program.

Short term solution: Simply build the fontcache with fc-cache32. Luckily, it seems we do not have other important software running on that server, that depends on the 64bit cache.

Long term solution: Upgrade the fontconfig libraries to a not so dead-old version. Newer versions of fontconfig have different cache files for 32bit and 64bit on the same installation.

November 11, 2010 · polesen · 3 Comments
Tags: , ,  Â· Posted in: Operating Systems, Programming