Arthropod-borne viruses contribute significantly to global mortality and morbidity in humans and animals. These viruses are mainly transmitted between susceptible vertebrate hosts by hematophagous arthropod vectors, especially mosquitoes. Recently, there has been substantial attention for a novel group of viruses, referred to as insect-specific viruses (ISVs) which are exclusively maintained in mosquito populations. Recent discoveries of novel insect-specific viruses over the past years generated a great interest not only in their potential use as vaccine and diagnostic platforms but also as novel biological control agents due to their ability to modulate arbovirus transmission. While arboviruses infect both vertebrate and invertebrate hosts, the replication of insect-specific viruses is restricted in vertebrates at multiple stages of virus replication. The vertebrate restriction factors include the genetic elements of ISVs (structural and non-structural genes and the untranslated terminal regions), vertebrate host factors (agonists and antagonists), and the temperature-dependent microenvironment. A better understanding of these bottlenecks is thus warranted. In this review, we explore these factors and the complex interplay between ISVs and their hosts contributing to this host restriction phenomenon.