ipkg update problem on Synology NAS

After upgrading my Synology NAS to DSM 4.0, ipkg stopped working. When I executed:

/opt/bin/ipkg update

it failed with this:

Downloading http://ipkg.nslu2-linux.org/feeds/optware/syno-e500/cross/unstable/Packages.gz
ipkg: /opt/ipkg-Rq65Jx/Packages.gz: No such file or directory
file_copy: ERROR: failed to copy /opt/ipkg-Rq65Jx/Packages.gz to /tmp/ipkg.ePVFQ7/cross.gz
An error ocurred, return value: 1.

after some debugging and googling, I found the cause to ipkg using a wrong version of wget. My system apparently has two wget binaries, one in /usr/syno/bin and one in /opt/bin. I need ipkg to grab the one in /opt/bin, like this in .profile:

PATH=...blah..what is was before..
PATH=/opt/bin:$PATH
export PATH

Reload .profile (or simply login and logout), and do a which wget to check you are getting the right one. Now proceed with ipkg update.

November 23, 2012 · polesen · Comments Closed
Tags: , , , ,  · Posted in: Tools

Converting RSA public key Modulus and Exponent into PEM file

Some days ago, I battled reading a raw RSA public key on iOS and using it to encrypt some data with on an iPad. Part of that battle was taking a “modulus” and “exponent” value from an RSA public key, and converting it into a PEM encoded public key file. I created this little C program to do this. It is using “openssl” and “libcrypto” to do its work.

In the case others, or I, some day need this, I created a gist for it here:

I’ve developed and tested this on OS X 10.7.4 with Xcode 4.3.2 installed. It can be built using the makefile which is also part of the gist:

I have no idea if this works properly on other platforms.

June 1, 2012 · polesen · One Comment
Tags: , , , ,  · Posted in: Programming

Titanium Module Development for iOS – Getting Started

Recently I had the “pleasure” of working with Titanium from Appcelerator. I needed to make a module for an iOS and Android library of a customer, so that developers could use the library from Titanium.

Basically, I followed this guide, but it showed to be incomplete or missing some information. This post is a follow-up on appcelerators own guide, which documents some of the gotchas and missing information, that I found out along the way.

I did all this on a Mac.

Step 1: Install Titanium Studio

Easy peasy (lemon squeezy). Just do as they say. I installed v1.8.1.

Step 2: Install SDKs

Then fire up the newly installed studio tool (aka: Eclipse), and have it install the mobile SDKs.

Step 3: Do the titanium alias thing

The guide mentions how to setup an alias, that runs the titanium.py script. That’s good advice.

One caveat here is: If you have something on the PATH variable which is also named titanium (as the alias also is named), Titanium Studio will try and execute that, instead of the alias. I had a directory named titanium on the PATH, and that’s not really that much of an executable.

Step 4: titanium create…

When the guide tells you to use titanium create... to create an initial module project, you should remember to avoid “-” (dash) in project name.

After the “create” command, you will now have an xcode project. Open up this project in xcode, open “Build settings” for the project, to be able to set TITANIUM_SDK variable. You can set it on 3 levels:

  • project level: ~/Library/Application Support/Titanium/mobilesdk/osx/$(TITANIUM_SDK_VERSION)
  • in target <yourmodulelib>: ~/Library/Application Support/Titanium/mobilesdk/osx/$(TITANIUM_SDK_VERSION)
  • in target “Build & Test”: $HOME/Library/Application Support/Titanium/mobilesdk/osx/$(TITANIUM_SDK_VERSION)

Note the use of $HOME in “Build & Test” target. This is because that target executes a python command in the shell, and it won’t expand tilde to homedir.

Check the binary build

The appcelerator guide mentions how to build.

BUT: It outputs a quadrillion lines when building, and when steps fail to build in the process, it will happily continue building, with the final result, that stuff is missing in the output artifact. And because of the humongous amount of output, you wont notice.

The appcelerator guide mention that you need markdown2 for documentation generation to work, and that bad things will happen if not. However, as mentioned, it wont stop from building, and the error will drown in all the other output from build. At first, you wont see any change, but the zip packaging will fail if you (like me) are missing both markdown2 and markdown.

Somewhere in all the build output, you will see:

DEBUG] Traceback (most recent call last):
[DEBUG] File "./build.py", line 216, in <module>
[DEBUG] package_module(manifest,mf,config)
[DEBUG] File "./build.py", line 193, in package_module
[DEBUG] docs = generate_doc(config)
[DEBUG] File "./build.py", line 66, in generate_doc
[DEBUG] import markdown
[DEBUG] ImportError: No module named markdown

which is due to this code in build.py failing:

        try:
                import markdown2 as markdown
        except ImportError:
                import markdown

it nicely catches if I don’t have markdown2, but has no catch if I don’t have markdown. Maybe I am the one in a million, that don’t even have markdown :-)

Also check, that the zip contains LICENSE and build.xcconfig, as that are some of the last files added in build process.

Other Gotchas

xcodebuild fail

When doing titanium run, it will continue running app, even though xcodebuild failed. Again, you might not notice due to all the build output.

frameSizeChanged

When developing module view classes, you are told to implement frameSizeChanged to adjust your view to its new bounds. The example also shows how to lazily construct your view, the first time frameSizeChanged is called. That is fine, just remember that frameSizeChanged never gets called, until you actually put some dimension on the view.

So, if you have this code in the titanium app that uses your module to create a view:

var view = yourlib.createYourView({
   "foo":42, 
   "bar":666
});

frameSizeChanged never will get called, which is turn means you won’t construct your view, which results in view proxies not forwarding messages to the view (because no view has been set).

If you change it to:

var view = yourlib.createYourView({
   "width":320,
   "height":480-20-44+1, // WTF: What's up with the one pixel off?
   "foo":42, 
   "bar":666
});

frameSizeChanged will be called.

Logging

To log output from objc code to titanium console, use: TiLogMessage(@"blah");

Xcode 4.3

Don’t upgrade yet (or be sure to keep the 4.2 version when upgrading) because it is not supported yet. Will change soon, I guess.

Javascript only modules

If you are developing javascript-only modules, you should correct TITANIUM_SDK in titanium.xcconfig, or else from compiler import Compiler will fail when building. (see TIMOB-7001).

And while we are at it. You cannot have both native (iOS or Android) AND javascript module code in same module.

I wanted to create a javascript part of my module, to provide a javascript enum-like thing, to support an objc enum type in the module API. So, I created an <mymoduleid>.js file and added my js code there. After compiling, it didn’t work. Hey, I thought, I will just get rid of the js module code and continue with my iOS native code. But then that code failed too. It was as if the js part was still there.

After a lot of cleaning in module project, module install places, build paths etc., I found out, that when doing javascript module development, that javascript file actually gets compiled into Classes/<yourmoduleid&gtModuleAssets.m::moduleAsset as a HEX string. Building actually goes in and modifies the source. Yikes! Cleaning that HEX string made my iOS module code come back to life.

Checking types and throwing exceptions

The module APIs are not type safe at all (not that surprising), so first thing you need to do in your module code, is checking types. If types fail to meet the needs, you need to report it somehow. This can be done with some exception throwing, like this:

  [self throwException:TiExceptionInvalidType 
             subreason:[NSString stringWithFormat:@"expected: %@, was: %@", [NSArray class], [args class]] 
              location:CODELOCATION];

Here I’m checking for NSArray type.

March 10, 2012 · polesen · 2 Comments
Tags: , ,  · Posted in: Programming

Xcode 4.2 + iOS5 building armv6+armv7 binary problems

After upgrading to Xcode4.2 and iOS5, we had problems building the binary for submission to AppStore. The final binary did not include both armv6 and armv7 architectures, so it was rejected. When linking, the linker told us about this warning:

warning: iPhone/iPod Touch: application executable is missing a required architecture.  
At least one of the following architecture(s) must be present: armv6 (-19033)

I think the use of this Xcode setting “$(ARCHS_STANDARD_32_BIT)” has changed its meaning since,..sometime. We solved this by doing this:

  • Goto Project->Build Settings
  • Find Architectures and change it from $(ARCHS_STANDARD_32_BIT) to two new values armv6 and armv7
  • While you are there, you can also check that Build Active Architecture Only has been set to No for the AdHoc and AppStore targets

October 30, 2011 · polesen · 3 Comments
Tags: ,  · Posted in: Tools

iOS Gotcha: Empty NSArray Optimization

I got bitten by a little iOS optimization the other day. It turns out, that NSArray has been optimized to represent the empty array with the same singleton object, no matter how instantiated.

This code:

NSArray *a1 = [[NSArray alloc] init];
NSArray *a2 = [[NSArray alloc] init];
NSLog(@"a1 == a2 => %d", (a1 == a2));

says:

a1 == a2 => 1

which indicates, that even though it looks like we instantiate 2 objects, the “a1″ and “a2″ variables point to the same object.

It doesn’t matter much how we instantiate the empty NSArray. We could also do:

NSArray *emptyArr = [[NSArray alloc] init];
NSArray *a1 = [NSArray arrayWithArray:emptyArr];
NSArray *a2 = [NSArray arrayWithArray:emptyArr];

// or

NSArray *a1 = [NSArray array];
NSArray *a2 = [NSArray array];

// or

NSArray *a1 = [NSArray arrayWithObjects:nil];
NSArray *a2 = [NSArray arrayWithObjects:nil];

I guess it is because NSArray is immutable, that this optimization was found okay to make. If I add just one object, I can’t reproduce it. So this:

NSArray *a1 = [NSArray arrayWithObject:@"foo"];
NSArray *a2 = [NSArray arrayWithObject:@"foo"];
NSLog(@"a1 == a2 => %d", (a1 == a2));

yields:

a1 == a2 => 0

and maybe more surprising, this code:

NSArray *a1 = [NSArray arrayWithObject:@"foo"];
NSArray *a2 = [NSArray arrayWithArray:a1];
NSLog(@"a1 == a2 => %d", (a1 == a2));

also yields:

a1 == a2 => 0

I would’ve guessed, that “[NSArray arrayWithArray:a1]” simply returned “a1″, because after all, NSArray is immutable. But it seems not to, when the given array isn’t an empty array.

Anyways, I got bitten because I had some code in a setter method with an optimization that went something like this:

if (_ivarArray != newArray) {
   // work the magic
}

and I really wanted to “work the magic”, even though “_ivarArray” previously also was an empty array.

October 28, 2011 · polesen · One Comment
Tags:  · Posted in: Programming

Dead Simple dos2unix “command”

The web is floating over (well, .. nearly) with various one-liners that can convert newlines of a text file from the dos/windows way to the unix way. These use tools like sed, tr, perl, .. to do the trick, and here is another one – using vim:

$ vi your-dos-newline-file.txt
:set fileformat=unix
:wq
and you are done. And I am pretty sure this can be one-lined too, with the proper v(h)im-sical command tooling :-)

October 10, 2011 · polesen · Comments Closed
Posted in: Operating Systems, Tools

Universal Library with both Simulator and iOS Architectures

Just a quick note to myself, because today I had to dig out from my brain, how it was I once built a Mach-O Universal library with both Simulator (i386) architecture and iOS (armv6+7) architectures in it.

xcodebuild -configuration Release -target my -sdk iphonesimulator4.0 clean build
xcodebuild -configuration Release -target my -sdk iphoneos4.3 clean build
lipo -create -output build/libmy.a build/Release-iphonesimulator/libmy.a build/Release-iphoneos/libmy.a

The final fat library created by “lipo” can now be used to link both Simulator and iOS builds.

When you construct the above lines for your project, a couple of xcodebuild commands that list info about a given xcode project, might come in handy:

xcodebuild -list yourProjectName
xcodebuild -showsdks

Where “-list” list information like “targets” and “build configurations” about given project and “-showsdks” shows the possible sdk names to use.

August 22, 2011 · polesen · Comments Closed
Tags: ,  · Posted in: Programming, Tools

Hibernate MultipleHiLoPerTableGenerator deadlock and TableGenerator.allocationSize

The other week I had the pleasure of visiting a former client, which had a problem with the connection pool being completely drained for connections, and as a consequence of that, their system pretty much locking up.

A dump of the stacks of all threads when the system locked up, told us the culprit was a specific web-service call, and all the stacks were dead in a call to hibernates MultipleHiLoPerTableGenerator.generate method. Like this:

38018:http-7555-140, state=BLOCKED
	org.hibernate.id.MultipleHiLoPerTableGenerator.generate(MultipleHiLoPerTableGenerator.java:204)
	org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:122)
	org.hibernate.ejb.event.EJB3PersistEventListener.saveWithGeneratedId(EJB3PersistEventListener.java:69)
	org.hibernate.event.def.DefaultPersistEventListener.entityIsTransient(DefaultPersistEventListener.java:179)
	org.hibernate.event.def.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:135)
	org.hibernate.event.def.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:61)
	org.hibernate.impl.SessionImpl.firePersist(SessionImpl.java:800)
	org.hibernate.impl.SessionImpl.persist(SessionImpl.java:774)
	org.hibernate.impl.SessionImpl.persist(SessionImpl.java:778)
	org.hibernate.ejb.AbstractEntityManagerImpl.persist(AbstractEntityManagerImpl.java:668)
	sun.reflect.GeneratedMethodAccessor553.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	java.lang.reflect.Method.invoke(Method.java:597)
	org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:365)
	$Proxy62.persist(Unknown Source)
	sun.reflect.GeneratedMethodAccessor553.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	java.lang.reflect.Method.invoke(Method.java:597)
	org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:240)
	$Proxy62.persist(Unknown Source)
	org.springframework.orm.jpa.JpaTemplate$5.doInJpa(JpaTemplate.java:264)
	org.springframework.orm.jpa.JpaTemplate.execute(JpaTemplate.java:183)
	org.springframework.orm.jpa.JpaTemplate.persist(JpaTemplate.java:262)

What’s interesting of this particular web-service call is, that it generates a lot of new rows of a specific JPA mapped entity in the system. Hence, it calls the id-generator a lot of times. And when the system is under heavy load, this particular web-service method is called often, which generates lots of parallel transactions, all trying to generate a lot of ids.

The class was mapped like this:

@Entity
public class DetailRow implements Serializable {

  private Long id;

  @Id
  @Column(unique = true, nullable = false)
  @GeneratedValue(strategy = GenerationType.TABLE, generator = "DetailRow.id")
  @TableGenerator(name = "DetailRow.id", table = "IdSequence", pkColumnName = "Name", valueColumnName = "WaterMark", pkColumnValue = "DetailRow", allocationSize = 1)
  public Long getId() {
    return id;
  }

  public void setId(Long id) {
    this.id = id;
  }
}

Now, it turns out others have had this problem too. The problem is with the MultipleHiLoPerTableGenerator of hibernate, which allocates new ids in a separate transaction (which it must, so that’s kinda okay). But, because it grabs an extra connection while holding an existing connection, it can deadlock when other calls do the same at the same time. The calls can end up hold the one connection all the others need. This is no news, and people have filed bugs (HB-1246 and HHH-1739) in the hibernate bug-db for it. None of them with a fix though.

I myself have no final fix for it. But I have an easy solution, which, at least for us, made the probability for the deadlock so small, that it stopped happening.

Solution

The use of @TableGenerator with hibernate as ORM provider, maps underneath to MultipleHiLoPerTableGenerator. I think we got that covered by now :-) But, there is an interesting parameter “allocationSize” to @TableGenerator. This makes the generator work in “chunks of ids”. Given the “IdSequence” table have a watermark of “123″, and we have set allocationSize to “1000″, the generator will increment watermark to “124″ ONCE and then for the next 1000 ids allocate 123000, 123001, 123002, ..123999 and first THEN, will it touch the “IdSequence” table again, to increment the watermark.

Setting “allocationSize” on @TableGenerator severely lowers the number of times an extra connection is needed, hence also making it a lot less probable, that it will end up in a deadlock. The “bug” is still there, because, technically, it still can happen. But practically, it won’t happen, if setting “allocationSize” high enough.

Caveat

Not many.

Remember that “allocationSize” is multiplied onto the “watermark” value, so to avoid a large unused hole of ids when deploying a new version with a larger “allocationSize”, you should divide the watermark with “allocationSize”.

Also, there can (will) be holes in the sequence, as unused ids from a “chunk” are lost when process dies. No big deal for us. Might be for some.

And what is a “high enough” value for “allocationSize” then? Well, it depends on your system. How many rows are created, and at which rate, and with how many concurrent transactions, related to your connection pool size. Do the math for loaded scenarios and try setting it to what you think is “high enough”.

August 15, 2011 · polesen · Comments Closed
Tags: , ,  · Posted in: Programming

Problems with require ‘mongo_mapper’ on cloudfoundry

I’ve been playing a bit with cloudfoundry today, and I wanted to do a simple example that connected to a mongodb service from ruby. Should be simple but I had some trouble. Some of the trouble I am sure stems from the fact that my ruby skills are somewhat rusty. Other from a somewhat alpha-like experience of the cloudfoundry product, which is supposed to be in beta :-)

I started out simple, with these ruby require lines in the top of my ruby sinatra app:

require 'sinatra'
require 'json'
require 'mongo'
require 'mongo_mapper'

Unfortunately, that made vmc update of my app fail. Cloudfoundry couldn’t start the app, and there were no logs or crashlogs to be seen :-( A little trial-n-error told me that it was the require of ‘mongo_mapper’, that made it fail. After some more trial-n-errors and a lot of googling, I could put together the needed bits.

This page at the cloudfoundry site has good info on what is required (and what they know won’t work) when running ruby apps on cloudfoundry. It also said, that I should be using Bundler and a Gemfile to describe my gem dependencies and package the app before deployment. So, I ended up with this Gemfile:

source "http://rubygems.org"
gem 'sinatra'
gem 'json'
gem 'mongo'
gem 'mongo_mapper'

and changed the require-lines in the sinatra application ruby file with this single line:

Bundler.require

When using bundler we then need to do these two steps before vmc push or update (each time dependencies have changed):

bundle package
bundle install

If you don’t have bundler, you can simply do:

sudo gem install bundler

June 10, 2011 · polesen · Comments Closed
Tags: , , ,  · Posted in: Programming

Springified Servlets, Listeners and Filters

Here are some short, but fully functional, templates for when you have a need for HttpServlet, ContextListener or Filter implementations, that also need spring bean dependencies.

You use them by subclassing them and calling the getBean method when a dependency is needed, like this:

// ..blah blah..inside subclass
MyService myService = getBean("myService");

Okay, here they come..

SpringHttpServlet

package net.techper.server.web;

import org.springframework.context.ApplicationContext;
import org.springframework.web.context.support.WebApplicationContextUtils;

import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;

public abstract class SpringHttpServlet extends HttpServlet {

  private ApplicationContext context;

  @Override
  public void init(ServletConfig config) throws ServletException {
    super.init(config);
    context = WebApplicationContextUtils.getRequiredWebApplicationContext(config.getServletContext());
  }

  @Override
  public void destroy() {
    context = null;
    super.destroy();
  }

  protected <T> T getBean(String beanName) {
    return (T) context.getBean(beanName);
  }
}

SpringContextListener

package net.techper.server.web;

import org.springframework.context.ApplicationContext;
import org.springframework.web.context.support.WebApplicationContextUtils;

import javax.servlet.ServletContext;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;

public abstract class SpringContextListener implements ServletContextListener {

  private ApplicationContext context;

  @Override
  public void contextInitialized(ServletContextEvent sce) {
    context = WebApplicationContextUtils.getRequiredWebApplicationContext(sce.getServletContext());
  }

  @Override
  public void contextDestroyed(ServletContextEvent sce) {
    context = null;
  }

  protected <T> T getBean(String beanName) {
    return (T) context.getBean(beanName);
  }
}

SpringFilter

package net.techper.server.web;

import org.springframework.context.ApplicationContext;
import org.springframework.web.context.support.WebApplicationContextUtils;

import javax.servlet.Filter;
import javax.servlet.FilterConfig;
import javax.servlet.ServletContext;
import javax.servlet.ServletException;

public abstract class SpringFilter implements Filter {

  private ApplicationContext context;

  @Override
  public void init(FilterConfig filterConfig) throws ServletException {
    context = WebApplicationContextUtils.getRequiredWebApplicationContext(filterConfig.getServletContext());
  }

  @Override
  public void destroy() {
    context = null;
  }

  protected <T> T getBean(String beanName) {
    return (T) context.getBean(beanName);
  }
}

March 5, 2011 · polesen · 2 Comments
Tags: , , , ,  · Posted in: Programming