The Judeans had begun to build the temple but local opposition and shrewd politics brought a letter from the office of Cambyses ordering public building to stop. (He's named Ahauserus and Artaxerxes-in Ezra 4:6-7 and he reigned from 530-522.) There's good reason to believe that the local critics had lied about the kind of building the Jews were building because king Cyrus the Great (father of Cambyses) had explicitly allowed Israel to rebuild the house of the Lord (Ezra 1:1-4). They probably made out that a fortress was being built though the truth is some temples were only a little less that a fortress. At a later date these local opponents will try to stop Nehemiah building my spreading the word he was declaring himself king (Nehemiah 6:5-7). In any case the Jews quit building--the temple that is.
But they went ahead and built their houses. Finished them too with panels (Haggai 1:4). God wasn't jealous of their houses, he didn't care if they'd been situated in Acapulco but he was jealous of where their hearts were situated and when the temple lay untouched for something like twelve years he spoke his piece (Haggai 1:2-4).
But it wasn't just how long the temple lay in ruins that troubled God because he was never particularly impressed with temples (see 2 Samuel 7:5-7); it was their continued excusing of themselves. "Oh, the work on the temple should be renewed, there's no doubt about that; but not right now," they said (1:2). And while they put it off they died a little each day. On the surface their delay makes sense but God must have known things that we don't so he spoke to them about it; besides, the Persian court was in an uproar that God had started (Haggai 2:6-7 and 20-23). At first God spoke to them through crop failure and the resultant inflation (1:6-7) but that didn't register so he called Haggai and Zechariah to have a word with them.
It's a bit ironic that they lived in the paneled houses but had little to eat. They had fine houses but their money was devalued. We need more than the house of our dreams, don't we? God knew he was what they needed so when he called them to build him a house (1:8) he wasn't sulking. He says he made life hard for them (it wasn't just "bad luck") to get them to turn to him (1:9-11).
What's life in the big paneled house when God sits homeless on the hill? And where did they get their houses? God built them for them; and they leave him sitting in the ruins of his? You understand it wasn't their building he wanted. He wanted their hearts and wanted them to show it by the building.
But when God calls us to worship it isn't self-centred or vanity. He's worthy of all our worship of course but we need to worship him more than he feels the need of our worship. Life in the big paneled house without a place to draw us together in worship and prayer and confession--that's life taking a nose-dive.