Contrary to common belief, victory points have nothing to do with matchmaking. According to the official definition:
Your opponents are selected from players with a similar matchmaking score. Every time you get a new Mercenary base on your map, either through exploration or invasion, your matchmaking score increases. Using ‘Find new opponent’, Headquarters level, experience level or Victory Points have no effect on your matchmaking score.
Matchmaking score will gradually decrease over time if you have several player opponents on your map. Therefore, if you are regularly matched with opponents you cannot beat, you will eventually get easier opponents. Only active players’ matchmaking score is adjusted down, so leaving the game for a long period of time won’t result in the game getting too easy”
Breaking that down, the only time your matchmaking score depends on your base is when being matched up on other peoples’ maps.