Very large and very complicated
The LLVM project has over 2 million lines, or 1.3 GB, of code. It's one of the largest projects on Github. It's so large that (at least at one point) you couldn't even push it to GitHub without workarounds.
With this, LLVM has a lot of unnecessary complexity. There are a lot of features your compiler isn't going to use unless it's C++, and a hobbyist or prototype compiler is definitely not going to use.
LLVM was created and is maintained by lots of talented developers, but nonetheless, a repo that large and complex is going to have issues. The following don't just apply to LLVM, but any large and complicated existing IR:
Steep learning curve: Because LLVM is so complex, it takes a lot of effort to understand how to use. See the reference for llvm::Value: there are so many methods and subclasses it's hard to figure out which ones you want and which ones to ignore. There are also many subtle details which can trip up beginners (though off the top of my head I can't think of any examples, I remember this was the case).
Lack of freedom: If you use LLVM as an IR you're practically required to adhere to LLVM conventions: e.g.
IRBuilder to construct values, using llvm
Metadata. If you use your own IR, you can implement techniques and optimizations which are hard or straight-up impossible in LLVM.
Framework problem (AKA more lack of freedom): It's not just in the IR. LLVM provides many generic types in the ADT module, including its own string and bit-vector; though you don't have to, it can be hard avoiding
BitVector in your own, non-IR code. More likely, you'll unintentionally write your AST and other earlier phases to be easier to translate into LLVM, which may make your language less unique more like C++. Like with most big frameworks, you could accidentally make your entire project revolve around using LLVM, especially if it's a small project and the majority of its code ends up being IR generation and LLVM integration.
Large dependency: LLVM is 1.3 GB in source, but its install build is about 12 GB. This can probably be stripped and minified for a language which only relies on IR and assembly generation, but still, LLVM is going to take up a lot of space in your app's distribution.
Bugs: In over 2 million lines of code there are definitely a lot of bugs. The GitHub repo has over 20,000 open issues.
Hard to debug: Because LLVM is so large, when there's an internal bug or even when there's not, it's hard to figure out where the bug originated and how to fix it, because you don't understand LLVM's inner workings.
As mentioned, these problems don't just apply to LLVM, but any other large general-purpose IR. If you write your own IR, you have control: you can make a small IR which only supports your language's features, and is written in your compiler's code style and design philosophy. This IR will be easier to understand, easier to integrate, easier to extend with features which would be challenging or impossible with LLVM, and easier to debug.
Of course this skips over the many benefits of LLVM and other popular IRs, and challenges using a custom IR, like:
Writing your own IR is hard
LLVM has a very good design philosophy and encourages good practices which may be better than the ones you would use
LLVM has plenty of general-purpose features and optimizations which would be hard to implement yourself
LLVM has a team of experts fixing bugs and constantly making improvements, you won't have to maintain the IR yourself if you use LLVM
and so on
But there are good reasons why you may want to write your own IR. And even if you do end up using LLVM, they are good issues to prepare for and avoid.